Image processing apparatus, image processing method, and image processing system

ABSTRACT

An image processing apparatus includes a data acquisition unit that acquires first sensing data that is image sensing data and second sensing data that is sensing data of a type different from the first sensing data and a display processing unit that performs a display process for displaying an augmented reality space formed by augmenting a real space with use of the first sensing data and the second sensing data. The display processing unit performs, by using the second sensing data, a display process for superimposing a virtual object on a real object recognized as an object present in the real space.

TECHNICAL FIELD

The present technology relates to a technical field of an imageprocessing apparatus, an image processing method, and an imageprocessing system for performing image processing associated with anaugmented reality space by using two types of sensing data.

BACKGROUND ART

A technology associated with an augmented reality space produced byarranging a virtual object in a real space is known.

For example, disclosed in the following PTL 1 is a technology whichpresents an augmented reality space to a user wearing a head-mounteddisplay.

CITATION LIST Patent Literature [PTL 1]

-   PCT Patent Publication No. WO2018/179644

SUMMARY Technical Problems

Described in PTL 1 is improvement of user-friendliness achieved bysuperimposed display of information indicating a waiting time, forexample. Further disclosed is such a technology which controls a sensoron the basis of required accuracy variable for each application.

Meanwhile, when a user experiences an augmented reality space, thisaugmented reality space provided for the user is required to be such aspace which presents information appropriate for the user in anappropriate mode.

The present technology has been developed in consideration of theabove-mentioned circumstances. It is an object of the present technologyto provide an augmented reality space achieving high user-friendlinessor an augmented reality space achieving high visibility.

Solution to Problems

An image processing apparatus according to the present technologyincludes a data acquisition unit that acquires first sensing data thatis image sensing data and second sensing data that is sensing data of atype different from the first sensing data, and a display processingunit that performs a display process for displaying an augmented realityspace formed by augmenting a real space with use of the first sensingdata and the second sensing data. A display process for superimposing avirtual object on a real object recognized as an object present in thereal space is performed by using the second sensing data.

The superimposed display of the virtual object on the real objectachieves display of an augmented reality space where the virtual objectis arranged without giving a sense of strangeness.

The second sensing data of the image processing apparatus describedabove may be sensing data from which distance information associatedwith the real object is extractable.

According to this configuration, the display mode of the virtual objectcan be varied according to the distance information.

The image processing apparatus described above may include a switchingprocessing unit that switches between a first mode for accepting a firstoperation for the virtual object and a second mode for accepting asecond operation for the virtual object. The switching processing unitmay switch to the first mode in a case where a predetermined portion ofa user is recognized within the augmented reality space by using thefirst sensing data or the second sensing data.

For example, a first process is executed in a case where a finger of theuser is recognized on the basis of the second sensing data from whichdistance information is extractable.

In the image processing apparatus described above, the first mode may bea mode for accepting an operation performed for the virtual objectaccording to movement of the predetermined portion in the augmentedreality space, and the second mode may be a mode for accepting a touchoperation performed by the user for a display region of the virtualobject on a display unit.

For example, in a case where a virtual touch operation performed by afinger of the user for the virtual object in the augmented reality spaceis detected by a ranging sensor or the like, transition to the firstmode is achieved. In a case where a touch operation for the image regionof the virtual object displayed on the display unit is detected,transition to the second mode is achieved.

The display processing unit of the image processing apparatus describedabove may designate a different display mode of a display object betweenthe first mode and the second mode.

The display object is an object displayed on the display unit. Forexample, the display object includes not only the virtual objectarranged in the augmented reality space, but also an operation iconimage and the like not arranged in the augmented reality space. Theexample which varies the display mode of the display object includes anexample which changes a display size, an example which switches betweendisplay and non-display for each display object, and other examples.

The display processing unit of the image processing apparatus describedabove may perform a display process for designating a different size ofthe virtual object between the first mode and the second mode.

For example, an optimum UI (User Interface) for the first mode may bedifferent from an optimum UI for the second mode. In this case, adifferent size of the virtual object is designated for the first modeand the second mode. Specifically, in a case where an operation for thevirtual object is detected in the augmented reality space, the virtualobject is displayed in a large size to improve visibility, for the user,of letter information or the like displayed as the virtual object.Meanwhile, in a case where a touch operation or the like for the displayunit is detected, this operation is recognized as a selection operationfor selecting the virtual object, and a process for displaying a webpage associated with the virtual object provided as an advertisementobject or the like is executed, for example.

The display processing unit of the image processing apparatus describedabove may perform an occlusion process for the virtual object on thebasis of distance information associated with the predetermined portion.

For example, in a case where the predetermined portion is a finger ofthe user and is located at a short distance, an occlusion process isperformed to hide the virtual object behind the finger of the user.

The image processing apparatus described above may include an operationtarget identification unit that identifies the virtual object designatedas a target of an operation performed by the predetermined portion, withuse of the first sensing data in a case where the predetermined portionis recognized.

It is possible that the user operates the virtual object while viewingthe virtual object and an image of his or her finger displayed on thedisplay unit. In such a case, the virtual object corresponding to theoperation target is identified according to a positional relationbetween the image of the finger and the virtual object on the displayunit.

The switching processing unit of the image processing apparatusdescribed above may switch to the second mode in a case where thepredetermined portion is detected on the basis of sensing data obtainedby imaging in a direction opposite to an imaging direction of an imagingunit that obtains the first sensing data.

In such a case, it is assumed that the predetermined portion of the useris unlikely to be detected in the first sensing data. Accordingly, it isconsidered that switching to the second mode for accepting the secondoperation is most appropriate.

The display processing unit of the image processing apparatus describedabove may perform a display process for issuing an alert regarding astep detected with use of the distance information.

In this manner, a notification of information associated with a step orthe like not recognized by the user can be issued as the virtual object.

The display processing unit of the image processing apparatus describedabove may determine the size of the virtual object to be superimposed onthe real object, according to a distance of the real object.

In this manner, the size of the virtual object is determined accordingto the distance from the user.

The second sensing data of the image processing apparatus describedabove may be sensing data on the basis of which a material of the realobject is identifiable. The display processing unit may perform adisplay process for superimposing the virtual object on the real objectwhose material is identified.

For example, the sensor for outputting the second sensing data is apolarization sensor, a multispectral sensor, or the like.

The image processing apparatus described above may include aself-position estimation unit that estimates a self-position. Theself-position estimation unit may correct the self-position on the basisof material information associated with the real object.

According to this configuration, highly accurate self-positioninformation can be obtained.

The display processing unit of the image processing apparatus describedabove may perform a display process for displaying the virtual object tobe superimposed, in a mode different between the real object whosematerial identified with use of the second sensing data is a transparentmaterial and the real object whose material identified with use of thesecond sensing data is not a transparent material.

For example, in a case where the first sensing data that is the imagesensing data is applied, there is a possibility that the real objectsuch as a door including glass and a window is difficult for the user torecognize. In such a case, the virtual object can be superimposed anddisplayed on a glass member on the basis of recognition of the realobject which is the glass member, with use of a polarization sensor, amultispectral sensor, or the like.

The second sensing data of the image processing apparatus describedabove may be sensing data on the basis of which a polarized state oflight reflected from the real object is identifiable.

Specifically, the second sensing data is sensing data output from apolarization sensor.

The display processing unit of the image processing apparatus describedabove may perform a display process for superimposing the virtual objecton the real object identified as a part of a human body with use of thesecond sensing data.

For example, some persons do not desire to appear in an image or avideo. Particularly, considering issues of privacy, it is not preferableto store or stream an image captured in a state where persons areidentifiable. According to this configuration, a part of a human body,such as a face, is identified, and the virtual object can besuperimposed and displayed on the identified part of the human body.

The first sensing data and the second sensing data of the imageprocessing apparatus described above may be generated on the basis ofsignals output from an identical sensor.

For example, not only image sensing data but also distance data can beacquired from data output from a CMOS sensor, in some cases.

An image processing method according to the present technology causes acomputer device to execute a process that acquires first sensing datathat is image sensing data and second sensing data that is sensing dataof a type different from the first sensing data, and a display processfor displaying an augmented reality space formed by augmenting a realspace with use of the first sensing data and the second sensing data.

An image processing system according to the present technology includesa display unit that is capable of displaying an image, a first sensingunit that acquires first sensing data that is image sensing data, asecond sensing unit that acquires second sensing data that is sensingdata of a type different from the first sensing data, and a displayprocessing unit that performs a display process for causing the displayunit to display an augmented reality space formed by augmenting a realspace with use of the first sensing data and the second sensing data.

The various effects described above can similarly be achieved by theimage processing method and the image processing system configured asabove.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram depicting a configuration example of asystem which acquires a three-dimensional model of a real space.

FIG. 2 is a block diagram depicting a configuration example of an imageprocessing apparatus according to a first embodiment.

FIG. 3 is a functional block diagram of an image processing unitaccording to the first embodiment.

FIG. 4 is a diagram depicting a display example of an augmented realityspace in a first case.

FIG. 5 is a diagram depicting a display example of an augmented realityspace in a second case.

FIG. 6 is a diagram depicting a display example of an augmented realityspace in a third case.

FIG. 7 is a diagram depicting a display example of an augmented realityspace in a fourth case and illustrating a state where a display processhas not yet been applied to human bodies.

FIG. 8 is a diagram depicting a display example of an augmented realityspace in the fourth case and illustrating a state where the displayprocess has been applied to the human bodies.

FIG. 9 is a flowchart for explaining a processing example executed by aprocessing unit according to the first embodiment.

FIG. 10 is a block diagram depicting a configuration example of an imageprocessing apparatus according to a second embodiment.

FIG. 11 is a functional block diagram of an image processing unitaccording to the second embodiment.

FIG. 12 is a diagram depicting a display example of a virtual objectaccording to the second embodiment.

FIG. 13 is a flowchart for explaining a processing example executed by aprocessing unit according to the second embodiment.

FIG. 14 is a diagram depicting a display example of an augmented realityspace when a first operation is conducted in a third embodiment.

FIG. 15 is a diagram depicting a display example of an augmented realityspace when a second operation is conducted in the third embodiment.

FIG. 16 is a functional block diagram of an image processing unitaccording to the third embodiment.

FIG. 17 is a flowchart for explaining a processing example executed by aprocessing unit in a fifth case.

FIG. 18 is an explanatory diagram depicting a state before a change of adisplay mode of an augmented reality space for accepting the firstoperation in the fifth case.

FIG. 19 is an explanatory diagram depicting a state after the change ofthe display mode of the augmented reality space for accepting the firstoperation in the fifth case.

FIG. 20 is an explanatory diagram depicting a state before a change of adisplay mode of an augmented reality space for accepting the secondoperation in the fifth case.

FIG. 21 is an explanatory diagram depicting a state after the change ofthe display mode of the augmented reality space for accepting the secondoperation in the fifth case.

FIG. 22 is a flowchart for explaining a processing example executed bythe processing unit in a sixth case.

FIG. 23 is a block diagram depicting a configuration example of an imageprocessing apparatus according to a fourth embodiment.

FIG. 24 is a flowchart for explaining a processing example executed by aprocessing unit according to the fourth embodiment.

DESCRIPTION OF EMBODIMENTS

Embodiments according to the present technology will hereinafter bedescribed in the following order with reference to the accompanyingdrawings.

-   -   <1. Augmented reality space>    -   <2. Configuration of image processing apparatus>    -   <3. Functional configuration>    -   <4. Example cases>    -   <4-1. First case>    -   <4-2. Second case>    -   <4-3. Third case>    -   <4-4. Fourth case>    -   <5. Process flow>    -   <6. Second Embodiment>    -   <7. Third Embodiment>    -   <7-1. Fifth case>    -   <7-2. Sixth case>    -   <8. Fourth Embodiment>    -   <9. Summary>    -   <10. Present technology>

1. Augmented Reality Space

An image processing apparatus according to the present technologyexecutes various processes associated with an augmented reality space(AR: Augmented Reality). An augmented reality space will herein betouched upon.

An augmented reality space is a space where a virtual object(annotation) is arranged in a real space imaged by an imaging unit. Asan example, an augmented reality space produced by arranging a virtualobject on a through image captured by an imaging unit included in aportable terminal device, such as a smartphone, is displayed on adisplay unit of this portable terminal device.

Arrangement of the virtual object is achieved by placing athree-dimensional model of the virtual object at a predeterminedposition. This arrangement includes a mode for superimposing a virtualobject on a real object.

For constructing an augmented reality space, a three-dimensional modelof a real space is created beforehand. For example, thethree-dimensional model of the real space is obtained by imaging atarget real space with use of an imaging device such as a celestialcamera. In this manner, determination of an arrangement mode of avirtual object is achievable. Alternatively, the three-dimensional modelmay be obtained by measurement with use of a measuring device other thana celestial camera, for example.

The three-dimensional model of the real space is stored in a serverdevice, or an image processing apparatus according to the presenttechnology, for example. FIG. 1 is an example where a three-dimensionalmodel of a real space, which is a shopping mall imaged by a celestialcamera 100, is stored in a server device 101. In addition to the above,the server device 101 also stores information associated with virtualobjects arranged in an augmented reality space. The informationassociated with the virtual objects contains information indicating adirection and a position of arrangement determined on the basis of thethree-dimensional model of the real space, three-dimensional models ofthe virtual objects, and the like.

The image processing apparatus according to the present technologyperforms a display process and other processes associated with theaugmented reality space described above. Described hereinafter will bean example of a smartphone 1 constituting the image processingapparatus.

The smartphone 1 is capable of executing a display process fordisplaying an augmented reality space according to an operation by auser, a process for detecting an operation associated with the augmentedreality space, and a process corresponding to a detected operation.

For example, the smartphone 1 superimposes display of logos of stores,store information, and the like created as virtual objects on display ofstores imaged as real objects. The user visually recognizes thesevirtual objects to obtain outlines or more detailed informationassociated with the stores.

Moreover, the smartphone 1 performs a process for displaying anaugmented reality space where characters as virtual objects are arrangedas concierges at an entrance and other places of the shopping mall. Theuser is allowed to acquire information associated with a store where adesired product can be purchased, information regarding a route to thecorresponding store, and the like by performing an operation for thecorresponding virtual object as a concierge (character object).

At this time, the smartphone 1 may acquire information associated with abehavior history or a purchase history of the user, and present, to theuser, presentation information selected on the basis of these items ofhistory information, via the virtual object as the concierge.Specifically, in a case of detection of a predetermined operationperformed by the user for the virtual object as the concierge, thesmartphone 1 executes a process for providing the user with informationassociated with a store suited for preferences of the user according tothe predetermined operation.

The smartphone 1 performs a process for displaying an augmented realityspace where virtual objects including translated letter information arearranged. By this process, the user is allowed to enjoy shopping on thebasis of translated information even in a foreign shopping mall or thelike. Note that the smartphone 1 achieves not only translation of letterinformation, but also arrangement and display of virtual objectsincluding translated letter information, without giving a sense ofstrangeness. In this manner, for example, the user is allowed tovisually recognize a poster or the like as a virtual object arranged inan appropriate direction according to a direction of a wall or the like.Accordingly, it is possible to improve a sense of immersion of the userinto the augmented reality space.

The smartphone 1 executes a process for arranging virtual objects or thelike for raising elation of the user. By this process, the user isallowed to enjoy shopping and the like more.

The smartphone 1 superimposes display of virtual objects includingadvertisement information on real objects such as a wall. The user istherefore allowed to acquire detailed advertisement information andpurchase a product displayed in an advertisement, by performing anoperation for the advertisement.

As described above, the smartphone 1 is capable of executing a displayprocess for displaying an augmented reality space, a process foraccepting an operation to achieve various processes corresponding to theoperation input from the user to the augmented reality space, and aprocess performed according to the operation. Accordingly, the user isallowed to enjoy various services by performing various operations forthe augmented reality space displayed on the smartphone 1.

2. Configuration of Image Processing Apparatus

FIG. 2 depicts a configuration of the smartphone 1 constituting theimage processing apparatus according to the first embodiment.

The smartphone 1 has a configuration capable of generating two types ofsensing data. Several examples of sensing data are adoptable as the twotypes of sensing data. In this example, captured image data and distancedata will be adopted by way of example.

The smartphone 1 includes a first sensing unit 2, a second sensing unit2, and an image processing unit 4.

The first sensing unit 2 includes an image sensor 5 which is a CCD(Charge Coupled Device) type or CMOS (Complementary Meta-OxideSemiconductor) type image sensor, for example, and an imaging opticalsystem 6 for guiding incident light to the image sensor 5. The imagingoptical system 6 includes various types of lenses such as a zoom lensand a focus lens and optical components such as a shutter mechanism.

The image sensor 5 exposes light entering from a subject via the imagingoptical system 6, and photoelectrically converts the exposed light toobtain an electric signal.

Pixels constituting sensing elements are two-dimensionally arrayed toform a sensor surface of the image sensor 5. Moreover, the image sensor5 includes a vertical drive unit, a horizontal drive unit, and the liketo achieve photoelectric conversion by the sensing elements.

The image sensor 5 includes a processing unit which performs processessuch as a CDS (Correlated Double Sampling) process, an AGC (AutomaticGain Control) process, and an A/D (Analog/Digital) conversion processfor an electric signal obtained through photoelectric conversion by thepixels. Accordingly, the image sensor 5 obtains captured image data asdigital data. The captured image data obtained by the image sensor 5 isoutput to the image processing unit 4 disposed in a subsequent stage.

The second sensing unit 3 includes a light emitting unit 7 and adistance sensor 8. Moreover, the second sensing unit 3 may include anoptical system for guiding light to the distance sensor 8.

The light emitting unit 7 performs a process for applying distancemeasurement IR (Infrared) light toward the outside of the smartphone 1.In a case where an iToF (indirect Time of Flight) system is used foracquiring distance data, the light emitting unit 7 includes a modulationcontrol unit which modulates intensity of irradiation light. Themodulation control unit supplies a driving signal at a low frequency(e.g., 70 MHz or lower) to a light source included in the light emittingunit 7, to apply intensity-modulated light from the light source. Thelight source of the light emitting unit 7 achieves pulsed light emissionwhich applies light having modulated intensity according to a supplieddriving signal having a low-frequency square wave.

Alternatively, distance data may be acquired with use of a dToF (directTime of Flight) system which achieves distance measurement on the basisof a time difference between irradiation timing of irradiation light andreception timing of reflection light.

The distance sensor 8 includes a sensor surface where pixelsconstituting sensing elements each having sensitivity to IR light aretwo-dimensionally arrayed. Moreover, the distance sensor 8 includes avertical drive unit, a horizontal drive unit, a transfer gate driveunit, and the like to achieve photoelectric conversion by the sensingelements.

The image processing unit 4 acquires captured image data as the firstsensing data from the first sensing unit 2. Moreover, the imageprocessing unit 4 acquires distance data as the second sensing data fromthe second sensing unit 3.

The image processing unit 4 acquires three-dimensional data and the likeof a virtual object VOB from an external processing device such as theserver device 101 described above.

The image processing unit 4 recognizes a real object ROB on the basis ofthe distance data obtained from the second sensing unit 3. Note that thereal object ROB may be recognized with use of not only the distance dataobtained from the second sensing unit 3, but also the captured imagedata obtained from the first sensing unit 2.

The image processing unit 4 executes processes for arranging andrearranging the virtual object VOB in the augmented reality space and aprocess for causing the display unit to display the augmented realityspace where the virtual object VOB is arranged.

The image processing unit 4 performs various types of display processesaccording to an operation by the user as well as the processes describedabove. For example, the image processing unit 4 performs a process fordisplaying a web page containing details of a product, in response to anoperation performed by the user to select an advertisement of thisproduct. Details of each of examples will be described below.

The smartphone 1 includes a control unit 9, a display unit 10, acommunication processing unit 11, and a memory unit 12.

The control unit 9 has an arithmetic processing function performed by aCPU (Central Processing Unit) or the like to achieve overall control ofthe smartphone 1. The control unit 9 transmits and receives informationto and from an external information processing device via thecommunication processing unit 11. For example, information associatedwith the virtual object VOB described above is transmitted to the imageprocessing unit 4 via the communication processing unit 11 and thecontrol unit 9.

Note that the image processing unit 4 and the communication processingunit 11 may be configured to directly exchange information with eachother.

The display unit 10 has a function of displaying captured image data.Moreover, the display unit 10 has a touch panel function to function asan operation unit. The control unit 9 is capable of executing a processfor detecting an input operation input from the display unit 10 providedas an operation unit and a process for handling the input operation.

The control unit 9 stores, in the memory unit 12, information necessaryfor performing various types of control.

The smartphone 1 includes a GNSS (Global Navigation Satellite System)unit 13 and a sound input unit 14.

The GNSS unit 13 has components such as a reception unit which receivesradio waves generated from a positioning system including multiplesatellites and the like.

The control unit 9 executes a process for estimating a self-position byusing radio waves received by the GNSS unit 13.

Note that the process for estimating the self-position of the smartphone1 may be performed without using information acquired from the GNSS unit13. For example, the smartphone 1 may acquire self-position informationby connecting to a wireless LAN (Local Area Network) such as Wifi(registered trademark).

The control unit 9 performs the process for estimating the self-positionon the basis of an acoustic signal input from the sound input unit 14.For example, in a certain case, a floor in a shopping mall or the likehaving multiple floors is difficult to identify on the basis ofself-position estimation using the GNSS unit 13. In such a case,detailed self-position estimation is achievable by acquiring, via thesound input unit 14, music, acoustic information, or the like generatedfrom surrounding stores or other places.

In addition, the smartphone 1 includes a sound output unit 15. The soundoutput unit 15 includes a speaker or the like, and is used for providingvoice navigation for the user, for example.

3. Functional Configuration

FIG. 3 is a block diagram depicting a functional configuration includedin the image processing unit 4 of the smartphone 1.

The image processing unit 4 has a data acquisition unit 21, arecognition processing unit 22, and a display processing unit 23.

The data acquisition unit 21 acquires captured image data (image sensingdata) as the first sensing data obtained from the image sensor 5 anddistance data (distance sensing data) as the second sensing dataobtained from the distance sensor 8. The data acquisition unit 21further acquires the virtual object VOB from a different informationprocessing device.

The recognition processing unit 22 performs a process for recognizing areal object within an angle of view on the basis of the captured imagedata and the distance data. Moreover, the recognition processing unit 22is capable of identifying a shape and the like of the real object on thebasis of the distance data.

Further, the recognition processing unit 22 may recognize a human byperforming a determination process for determining whether or not a realobject is a person or a part of a body.

The display processing unit 23 executes a process for causing thedisplay unit 10 to display an augmented reality space. Note that thedisplay processing unit 23 displays several types of objects on thedisplay unit 10. For example, an object present in the augmented realityspace, such as information associated with the real object ROB displayedon the basis of captured image data and with the virtual object VOBacquired from the different information processing device, is defined asone of targets of the display process performed by the displayprocessing unit 23.

Moreover, an icon image simply displayed on the display unit 10 otherthan the object present in the augmented reality space is also definedas one of targets of the display process performed by the displayprocessing unit 23. In the following description, a display item otherthan the real object ROB and the virtual object VOB present asthree-dimensional objects in the augmented reality space will bereferred to as an “icon image Pi.” In addition, the real object ROB, thevirtual object VOB, and the icon image Pi will collectively be referredto as a display object.

Shapes, sizes, and the like of the real object ROB and the virtualobjects VOB are variable according to changes of an imaging position andan imaging direction. However, a display mode of the icon image Pi isnot variable even if the imaging position or the imaging directionchanges. The display mode of the icon image Pi may be configured to bevariable by a factor other than those factors.

Note that the virtual object VOB displayed by the display processingunit 23 may be either a personalized object or a fixed object notvariable for each user. For example, the virtual object VOB such as asignboard representing a type of a store may be a fixed object notvariable for each user, while the virtual object VOB for presenting arecommended menu of a store may be personalized on the basis ofinformation indicating preferences of each user.

4. Example Cases

Respective example cases of the processes executed by the smartphone 1will be described.

<4-1. First Case>

A first case causes the display unit 10 to display an augmented realityspace where the virtual object VOB for alerting the user is arranged.

FIG. 4 depicts an example of this case. FIG. 4 depicts a part of thedisplay unit 10 of the smartphone 1 carried by the user walking along aroadside, and illustrates a state where an augmented reality space isdisplayed on the display unit 10.

The one virtual object VOB is displayed on the display unit 10 togetherwith the real objects ROB representing garden plants on a sidewalk andobjects present on the opposite side of a roadway.

The virtual object VOB is an object for alerting the user to thepresence of a step near the user, and is superimposed on a step portionlocated on the boundary between the roadway and the sidewalk.

Such display is achievable on the basis of distance data output from thedistance sensor 8 included in the second sensing unit 3. In some cases,the presence or absence of a step is difficult to identify on the basisof captured image data output from the image sensor 5. Even in such acase, the presence or absence of a step can appropriately be determinedon the basis of the distance data.

<4-2. Second Case>

A second case displays information provided for the user as informationsuperimposed on the real object ROB such as a wall.

An example depicted in FIG. 5 is an example where virtual objects VOBrepresenting posters each including an advertisement are superimposed ona wall. The virtual objects VOB are superimposed not only simply withina region where the wall is displayed, but also at locations according tothe direction of the wall in such a manner as not to give a sense ofstrangeness, i.e., in a manner as if real objects were actually presentin that area.

Such display mode is achievable on the basis of distance data outputfrom the distance sensor 8 included in the second sensing unit 3.Specifically, the wall direction is identifiable on the basis ofdistance information obtained from the distance sensor 8. Accordingly,the virtual objects VOB can be arranged in appropriate directionscorresponding to the wall direction.

Such display process allows the user to appreciate the virtual objectsVOB in a natural manner without feeling a sense of strangeness.

<4-3. Third Case>

A third case changes a display mode of the virtual objects VOB on thebasis of distance data. As an example, the sizes of the virtual objectsVOB on the display are changed on the basis of distance data.

FIG. 6 depicts an augmented reality space displayed on the display unit10 of the smartphone 1, and illustrates a state where roadside storesform a line on each of roadsides. Images of the respective stores aredisplayed on the basis of the real objects ROB. Among the real objectsROB on the right side of the road, the real object ROB located on thefront side is assumed to be a real object ROB1, and the real object ROBlocated on the rear side is assumed to be a real object ROB2. Inaddition, among the real objects ROB on the left side of the road, thereal object ROB located on the front side is assumed to be a real objectROB3, and the real object ROB located on the rear side is assumed to bea real object ROB4.

Moreover, FIG. 6 depicts a virtual object VOB1 which imitates a logo ofa store representing the real object ROB1, and is arranged as asignboard hung at a store front. Similarly, a virtual object VOB2imitates a logo of a store representing the real object ROB2, and isarranged as a signboard.

On the basis of distance data output from the distance sensor 8 of thesmartphone 1, it is identified that the real object ROB1 is locatedbefore the real object ROB2.

Accordingly, the virtual object VOB1 associated with the real objectROB1 is displayed in a size larger than the virtual object VOB2associated with the real object ROB2.

As described above, the display processing unit 23 of the smartphone 1performs the display process which increases the size of the display ofthe virtual object VOB superimposed on the real object ROB located onthe front side. This process allows the user to intuitively obtain acorrect sense of distance to the virtual object VOB. Specifically, thisprocess achieves presentation of the virtual object VOB without giving asense of strangeness to the user.

<4-4. Fourth Case>

A fourth case performs a display process for the specific real objectsROB by using the virtual objects VOB in a case where the specific realobjects ROB are identified.

Specifically, described will be a case which superimposes the virtualobjects VOB in a case where persons or parts of persons are identified.

For example, suppose that a different user not desiring to appear in thescreen of the smartphone 1 is present in this screen in a case where theuser of the smartphone 1 is visually recognizing an augmented realityspace via the smartphone 1. The different user does not desire to appearin the screen because there is a possibility that the user visuallyrecognizing the augmented reality space is recording a video ordistributing a recorded video in real time. For giving consideration tothe user not desiring to appear in a video or the like, the smartphone 1performs a process for achieving superimposed display of the virtualobjects VOB in such a manner as to prohibit identification of persons onthe basis of corresponding portions in a case where humans or parts ofthe bodies of the humans are detected within the augmented realityspace.

As an example, the virtual object VOB imitating a character image, suchas a dog and a cat, is superimposed on a face portion of a person.Alternatively, a state where image data corresponding to the faceportion of the person is blacked out or a state of mosaic processingapplied to the face portion may be displayed.

In addition, considering a possibility that a person is identifiable onthe basis of his/her hand or leg as well as his/her face, a similardisplay process may be performed for some parts of the body such asexposed hands and legs.

Further, considering a possibility that a person is identifiable on thebasis of his/her clothes or the like, a similar display process may beperformed for a whole region containing a human body regardless ofwhether or not each portion is covered with clothes.

FIG. 7 depicts a state of the display unit 10 of the smartphone 1, thedisplay unit 10 displaying an augmented reality space in which theabove-described processing has not yet been performed for parts of humanbodies. Meanwhile, FIG. 8 depicts a state of the display unit 10 of thesmartphone 1, the display unit 10 displaying an augmented reality spacewhere parts of the human bodies are blacked out.

Whether or not the real object ROB is a part of a human body can bedetermined on the basis of distance data output from the distance sensor8 included in the second sensing unit 3. Specifically, the shape of thereal object ROB is identifiable on the basis of the distance data.Thereafter, whether or not the real object ROB is a part of a human bodycan be determined by calculating a degree of agreement between theidentified shape of the real object ROB and shape data associated withfaces, hands, legs, or other portions of bodies of humans and stored inthe smartphone 1 or the server device 101.

While presented in this example has been the example of the secondsensing unit 3 including the distance sensor 8, a similar determinationprocess can be achieved by a thermosensor provided as the second sensingunit 3. Specifically, whether or not the real object ROB is a part of ahuman body can be determined on the basis of comparison between atemperature distribution or the like of the real object ROB and storedstandard data. In addition, the real object ROB imitating a shape of ahuman, such as a mannequin, can be appropriately determined as an objectnot corresponding to a part of a human body on the basis of temperatureinformation.

Note that the smartphone 1 operated by the user may execute anotification process for issuing a notification to the different usernot desiring to appear within an angle of view of the smartphone 1. Forexample, in a case where the smartphone 1 displays an augmented realityspace while imaging a real space, the control unit 9 may execute aprocess for lighting an LED (Light Emitting Diode) included in thesmartphone 1 in a specific light emission pattern or light emissioncolor.

In this manner, the different user that is present around the user iscapable of moving out of the angle of view of the smartphone 1 toprotect privacy.

Note that the light emission process described above may be performed onthe basis of distance data obtained from the distance sensor 8. Forexample, this light emission process may be executed in a case where thereal object ROB located within a predetermined distance (e.g., fivemeters) can be identified as a part of a human body. Accordingly, thenecessity of executing the light emission process is eliminated in acase where the different user is detected but located far away. Hence,reduction of power consumption of the smartphone 1 is achievable.

5. Process Flow

Described will be a flowchart executed by the image processing unit 4 orthe control unit 9 for practicing the first case to the fourth casedescribed above.

Note that the image processing unit 4 and the control unit 9 willcollectively and simply be referred to as a “processing unit” herein.

In Step S101 illustrated in FIG. 9 , the processing unit acquirescaptured image data from the image sensor 5 of the first sensing unit 2.

In Step S102, the processing unit acquires distance data from thedistance sensor 8 of the second sensing unit 3.

In Step S103, the processing unit generates display data as data to bedisplayed on the display unit 10, on the basis of the captured imagedata.

In Step S104, the processing unit determines whether or not the realobject ROB designated as a target of superimposition by the virtualobject VOB is present, on the basis of the distance data. Note that thisdetermination process may be achieved with use of either only thecaptured image data or both the captured image data and the distancedata.

The process in Step S104 will specifically be explained. It isdetermined whether or not a step is present around the user, on thebasis of the distance data acquired from the distance sensor 8 in thefirst case, for example. Moreover, steps designated as targets amongmultiple steps around the user may be narrowed down on the basis of thedistance data. Specifically, a step determined to be located within apredetermined distance from the user may be selected as a target ofsuperimposition by the virtual object VOB.

Moreover, the determination process in the second case is a process fordetermining whether or not a wall, a ceiling, a floor, or the likeavailable as an optimum place for display of an advertisement ispresent. For example, this process may determine whether or not a wallor the like allowing superimposition by the virtual object VOB at aneasily viewable angle for the user is present, on the basis of thedistance data. Moreover, this process may exclude a wall where the realobject ROB such as a poster is already attached or a ceiling where afluorescent light, an air conditioner, or the like is disposed.

Further, the determination process in the third case is a process fordetermining whether or not a store considered to be suitable for displayof a signboard or the like as the virtual object VOB is located withinan angle of view. The store considered to be suitable for display of asignboard or the like is a store in business, a store meeting preferenceinformation associated with the user, or a store located within apredetermined distance, for example. Moreover, a store which has usefulinformation for the user, such as time sale information, may bedesignated.

Finally, the determination process in the fourth case is a process fordetermining whether or not a face of a person is present within an angleof view. Note that, as described above, this process may determinewhether or not a face in a direction of a predetermined angle ispresent, or whether or not a face located within a predetermineddistance is present. Moreover, this process may determine whether or nota part of a human body including a face is present.

In a case of determining in Step S104 that the target real object ROB ispresent, the processing unit acquires or generates the virtual objectVOB in Step S105.

For example, in the first case, the processing unit acquiresthree-dimensional model information associated with the virtual objectVOB having a balloon shape for a step alert.

Moreover, in the second case, the processing unit acquiresthree-dimensional model information associated with a poster, andacquires letter information to be superimposed on the poster from anadvertisement server or the like. Thereafter, the processing unitcombines the letter information with the three-dimensional model togenerate the virtual object VOB.

Further, in the third case, the processing unit acquiresthree-dimensional model information associated with the virtual objectVOB imitating a signboard of a store.

In the fourth case, the acquisition process in Step S105 need not beperformed depending on cases. If a face or a part of a human body isblacked out as described above, the process in Step S105 is unnecessary.Alternatively, if the virtual object VOB is arranged in such a manner asto cover a face of a detected human with a mask imitating a face of ananimal as a different mode of the fourth case, the processing unitacquires three-dimensional model information associated with the virtualobject VOB representing the mask.

In the manner described above, the processing unit acquires informationor performs a generation process associated with the virtual object VOBobtained from a different information processing device. Note thatexecution of the generation process is not necessarily required.

Information indicating a current position of the user, a current time,or the like may be added to the virtual object VOB to be acquired or thevirtual object VOB to be generated herein. For example, for a spot wherea lot of traditional buildings are located, such as Kyoto, the virtualobject VOB in a mode not damaging an atmosphere of that spot (e.g.,Japanese taste object) may be acquired or generated.

Moreover, in a case where the current time is nighttime, the virtualobject VOB imitating a neon signboard may be acquired or generated.

In Step S106, the processing unit determines a display position, adisplay angle, a size, and the like of the virtual object VOB. Forexample, for an advertisement poster superimposed on a wall or the like,the processing unit determines the display angle of the virtual objectVOB in consideration of the direction or the like of the wall to producea view as if the poster were attached to the wall. Moreover, for athree-dimensional model signboard or the like, the processing unitdetermines the display size of the virtual object VOB such that the sizeincreases as the distance to the corresponding store decreases.

Note that the size of the virtual object VOB to be determined is notlimited to the size of the virtual object VOB on the display, i.e., thesize of an image area occupied by the virtual object VOB on the displayunit 10, and may be an absolute size of the three-dimensional model ofthe virtual object VOB. In addition, in this case, the sizes of thethree-dimensional models of the virtual objects VOB of the respectivestores may be equalized regardless of the distances to the stores. Whenthe sizes of the three-dimensional models are equalized, thethree-dimensional model located closer is necessarily displayed in alarger size.

Moreover, if the process for blacking out a face of a person isperformed in the fourth case, the process in this step is a process foridentifying an image area containing the face.

In Step S107, the processing unit performs a process for updatingdisplay data. For example, this process executes an occlusion processfor the real object blocked by the virtual object VOB.

In addition, a shading process considering an entrance quantity of reallight may be executed.

In Step S108, the processing unit performs a display control process.This process causes the display unit 10 to display an image of the realobject ROB on which the virtual object VOB is superimposed.

Note that, in a case where the real object ROB as a target ofsuperimposition by the virtual object VOB is determined, in Step S104,to be absent, a display control process for the real object ROB, i.e., aprocess for displaying what is generally called a through image on thedisplay unit, is executed in Step S108.

6. Second Embodiment

Described will be an example of a smartphone 1A in a second embodimentwhich is different from the first embodiment in that the second sensingunit 3 has a sensor different from the distance sensor 8.

A second sensing unit 3A according to the second embodiment includes apolarization sensor 31 as depicted in FIG. 10 . In addition, unlike inthe first embodiment, the second sensing unit 3A does not include thelight emitting unit 7.

The polarization sensor 31 is a mode of a sensor capable of identifyinga material of the real object ROB. Specifically, in a case where thereal object ROB including a glass material or the like is present, thepolarization sensor 31 outputs, as second sensing data, polarizationdata on the basis of which glass as the material of the real object ROBis identifiable.

As depicted in FIG. 11 , an image processing unit 4A of the smartphone1A includes the data acquisition unit 21, the recognition processingunit 22, the display processing unit 23, a material identification unit24, and a self-position estimation unit 25.

Note that the self-position estimation unit 25 may be eliminated as willbe described below.

Processes performed by the data acquisition unit 21, the recognitionprocessing unit 22, and the display processing unit 23 are similar tothe processes performed by the data acquisition unit 21, the recognitionprocessing unit 22, and the display processing unit 23 in the firstembodiment, and therefore will not be repeatedly described in detail.

As described above, the material identification unit 24 is capable ofidentifying the real object ROB including the glass material, on thebasis of the polarization data as the second sensing data.

In a case where the real object ROB is identified as a glass door, thedata acquisition unit 21 acquires information associated with thevirtual object VOB, such as an advertisement, superimposed on the glassdoor.

The display processing unit 23 performs a display process considering adisplay mode of the virtual object VOB on the basis of identifiedmaterial information. As an example, in a case where the real object ROBis identified as a glass door, the display processing unit 23 performs aprocess for displaying the virtual object VOB, such as an advertisement,to be superimposed on the glass door in a semitransparent state (seeFIG. 12 ). This process achieves display allowing visual recognition ofthe real object ROB located on the opposite side of the glass doorthrough the virtual object VOB and the real object ROB.

The self-position estimation unit 25 performs a process for estimating aposition of the smartphone 1 as a self-position. For example, theself-position estimation unit 25 identifies a rough position by usingposition information acquirable from the GNSS unit 13 included in thesmartphone 1A, and estimates further detailed position information onthe basis of the material information associated with the real objectROB and identified by the material identification unit 24.

For example, it is assumed that a floor where the smartphone 1A islocated in a building is not identifiable even with use of the positioninformation acquirable from the GNSS unit 13. Accordingly, detailedposition information associated with location of the smartphone 1A isestimated (corrected) on the basis of information indicating a position,a size, a shape, and the like of the real object ROB including the glassmaterial. In a case where the real object ROB including the glassmaterial and having a specific size is present only on a specific floor,or in a case where the real object ROB including the glass material andhaving a specific shape is present only on a specific floor, the floorof the smartphone 1A is identifiable on the basis of these items ofinformation indicating the real objects ROB.

In the manner described above, the smartphone 1A is allowed to identifythe floor which is not identifiable on the basis of information obtainedfrom the GNSS unit 13. Note that the material information need not bereferred to in a case where the floor is identifiable on the basis ofonly the position, the size, or the shape of the real object ROB.

A flow of a process executed by a processing unit (the image processingunit 4, the control unit 9, and the like) according to the secondembodiment will be described with reference to FIG. 13 . Note thatprocesses similar to the processes in the first embodiment are givenidentical reference signs, and will not be repeatedly described whereappropriate.

The processing unit acquires captured image data by executing a processin Step S101. Thereafter, the processing unit acquires polarization dataas the second sensing data from the polarization sensor 31 in Step S110.

By executing the respective processes in Steps S103 and S104, theprocessing unit generates display data on the basis of the capturedimage data, and determines whether or not the real object ROB requiringsuperimposition by the virtual object VOB is present.

At this time, in a case where the estimation process for estimating theself-position has been executed by the self-position estimation unit 25,the estimated self-position information is referred to for determiningin Step S104 whether or not the real object ROB requiringsuperimposition by the virtual object VOB is present.

For example, in a case of presenting to the user such informationassociated with a store A located on a second floor of a shopping mallwhere the user is currently shopping, it is possible that informationindicating that the store A is located on the second floor issuperimposed as the virtual object VOB. For this purpose, it is possiblethat a notification also containing a route for going up (down) to thesecond floor is issued. However, in a case where this user is alreadylocated on the second floor, such a manner of information presentationis unnecessary.

At this time, in a case where the fact that the user is already locatedon the second floor is recognized on the basis of the self-positionestimation process performed by the self-position estimation unit 25,information indicating that the store A is located on the second floor,a guide to a position of an elevator used for going down to the secondfloor, and the like are unnecessary. In this case, a route from acurrent place to the position of the store A (i.e., two-dimensionalroute) is only required to be presented.

Accordingly, letter information displayed with the presented virtualobject VOB can be reduced, and therefore, information presentation easyto read for the user is achievable.

In a case where the corresponding real object ROB is absent, theprocessing unit advances the process to Step S108 to perform a processfor causing the display unit 10 of the smartphone 1A to display athrough image.

On the other hand, in a case where the corresponding real object ROB ispresent, the processing unit advances the process to Step S105 toacquire or generate the virtual object VOB.

In Step S111, the processing unit determines a display mode of thevirtual object VOB on the basis of the polarization data. In this step,in the example described above, the processing unit performs adetermination process for determining whether or not to display in asemitransparent state, for example. Note that it is possible that asimilar process is carried out in a case where the virtual object VOB issuperimposed on the real object ROB including a semitransparent material(e.g., transparent acrylic plate) other than the glass material.

The processing unit determines a display position, a display angle, asize, and the like of the virtual object by executing the process inStep S106.

The processing unit executes a process in Step S107 to perform a processfor updating display data generated in Step S103. This update processgenerates display data for displaying an augmented reality space inwhich the virtual object VOB is arranged with the display modedetermined in Step S111, and the display position, the display angle,the size, and the like determined in Step S106 being reflected.

Note that the second sensing unit 3 in the second embodiment may includea multispectral sensor instead of the polarization sensor 31.

The multispectral sensor is a sensor which outputs multispectral imagedata on the basis of which spectrum reflection characteristics of thereal object ROB are recognizable.

Even in this case, the material of the real object ROB is similarlyidentifiable on the basis of the second sensing data.

Specifically, the spectrum reflection characteristics of the real objectROB vary according to the material, the color, or the like of the realobject ROB. Material information and color information associated withthe real object ROB can be obtained by matching between these variouscharacteristics and information output from the multispectral sensor.

In addition, the display processing unit 23 may determine the displaymode of the virtual object VOB to be superimposed, on the basis of thematerial information associated with the real object ROB.

For example, in a case where the real object ROB is a wooden wall, adisplay mode to be adopted is such a mode where the virtual object VOBas an advertisement poster is affixed to the wall via thumbtacks. In acase where the real object ROB is a glass wall, a display mode to beadopted is such a mode where the virtual object VOB as an advertisementposter is affixed to the wall via tapes. In a case where the real objectROB is a metal wall, a display mode to be adopted is such a mode wherethe virtual object VOB as an advertisement poster is affixed to the wallvia magnets.

By changing the display mode in the manner described above, display ofthe virtual object VOB full of reality is achievable.

Note that, while the first sensing data and the second sensing data areacquired from the first sensing unit 2 and the second sensing unit 3,respectively, in the first embodiment and the second embodiment, thesmartphone 1 (1A) may be configured to acquire the two types of sensingdata from one sensing unit.

Specifically, the image sensor 5 of the first sensing unit 2 may beconfigured to output distance data as the second sensing data as well ascaptured image data as the first sensing data. The image sensor 5 iscapable of outputting a signal on the basis of which an amount ofdefocus using what is generally called a contrast method can becalculated according to movement of a focus lens included in the imagingoptical system 6. In addition, the defocus amount is dependent on adistance to a subject. Accordingly, the image sensor 5 is considered tobe capable of outputting second sensing data on the basis of whichdistance data can be calculated.

Moreover, material information is acquirable for each of the realobjects ROB if inference using deep learning is made on the basis ofcaptured image data output from the image sensor 5 of the first sensingunit 2. Accordingly, the captured image data may be considered asinformation on the basis of which material information is identifiable.The image sensor 5 in this case may be considered as a sensorfunctioning as the first sensing unit 2 capable of outputting capturedimage data as the first sensing data, and also functioning as the secondsensing unit 3 capable of outputting data on the basis of which thematerial is identifiable as the second sensing data.

Note that, if inference using deep learning is made, captured image dataoutput from the image sensor 5 of the first sensing unit 2 may beconsidered as the second sensing data on the basis of which distancedata can be calculated. Specifically, distance data for each of the realobjects ROB can be extracted by performing a process for inferring eachdistance to the real objects ROB on the basis of the position, the size,or the like of each of the real objects ROB in a target image based onthe captured image data.

7. Third Embodiment

In a third embodiment, the smartphone 1 having the configurationdepicted in FIG. 2 is configured to perform two types of operations forthe virtual object VOB. However, as will be described below, thesmartphone 1 in the third embodiment includes an image processing unit4B.

This embodiment will specifically be described with reference to theaccompanying drawings.

FIG. 14 depicts an operation mode of a first operation included in twotypes of operations acceptable by the smartphone 1 according to thethird embodiment.

FIG. 14 depicts a state where the smartphone 1 is capturing images ofthree virtual objects VOB3, VOB4, and VOB5 that each have a poster shapeand are arranged in an augmented reality space. In addition, each of thevirtual objects VOB3, VOB4, and VOB5 not present in a real space isindicated by a dotted line. However, the virtual objects VOB3, VOB4, andVOB5 are each indicated by a solid line on the display unit 10 of thesmartphone 1 displaying the augmented reality space where the virtualobjects VOB are arranged.

As depicted in the figure, the first operation is an operation forselecting the virtual object VOB according to movement of apredetermined portion of the user in the augmented reality space. Anindex finger IF of the user is adopted herein as an example of thepredetermined portion of the user.

Specifically, the index finger IF in the augmented reality space movesaccording to movement of the index finger IF by the user in the realspace. Thereafter, the one virtual object VOB4 is selected according tomovement of the index finger IF in the augmented reality space. Thisseries of movement constitute the first operation. In other words, thefirst operation is an operation for allowing the user to feel as if thefinger directly touched the virtual object VOB that is arranged in theaugmented reality space but that is difficult to directly touch inreality. In addition, this operation is also considered to be anoperation for bringing an image region of the index finger IF displayedon the display unit 10 into contact with an image region containingdisplay of the virtual object VOB.

Note that the first operation may select multiple virtual objects VOBinstead of one virtual object VOB.

The virtual object VOB selected by the first operation of the user isidentified on the basis of captured image data output from the imagesensor 5. The user moves the index finger IF of the user in such amanner as to touch the virtual object VOB displayed on the display unit10 of the smartphone 1. Accordingly, it is preferable that the virtualobject VOB be identified on the basis of the captured image data.

FIG. 15 depicts an operation mode of a second operation as one of thetwo types of the operations acceptable by the smartphone 1 according tothe third embodiment.

As depicted in the figure, the augmented reality space where the virtualobjects VOB3, VOB4, and VOB5 are arranged is displayed on the displayunit 10 of the smartphone 1. The user performs a touch operation fortouching the display unit 10 functioning as an operation unit, in such amanner as to touch the virtual object VOB4 displayed on the display unit10. This touch operation constitutes the second operation.

By achieving at least two types of the operations described above, i.e.,the first operation for allowing the user to feel as if the fingertouched the virtual object VOB and the second operation as the touchoperation for touching the display unit 10, user friendliness improves.Specifically, ease of operations of the smartphone 1 is variableaccording to a position of the smartphone 1, a distance to the user, andsurrounding sceneries and circumstances. The present technologycontributes to improvement of operability for the user by practicing theabove two operations.

Several types of operations are adoptable as the touch operation. Forexample, a single tap operation, a double tap operation, a swipeoperation, a flick operation, a scroll operation, a drag operation, apinch-in operation, and a pinch-out operation are adoptable.

In the following description, a mode for accepting the first operationwill be referred to as a first mode, while a mode for accepting thesecond operation will be referred to as a second mode.

FIG. 16 depicts a functional configuration of the image processing unit4B included in the smartphone 1 which accepts the first operation andthe second operation.

As depicted in the figure, the image processing unit 4B includes thedata acquisition unit 21, the recognition processing unit 22, thedisplay processing unit 23, a switching processing unit 26, and anoperation target identification unit 27.

The data acquisition unit 21 acquires captured image data as the firstsensing data from the first sensing unit 2 and distance data as thesecond sensing data from the second sensing unit 3.

The recognition processing unit 22 performs a process for recognizingthe real object ROB within an angle of view on the basis of the capturedimage data and the distance data. Moreover, the recognition processingunit 22 is capable of identifying a shape and the like of the realobject ROB on the basis of the distance data.

Further, the recognition processing unit 22 according to the presentembodiment performs a process for recognizing the index finger IF of theuser. Either the captured image data or the distance data is availablefor performing the process for recognizing the index finger IF of theuser.

The display processing unit 23 executes a process for causing thedisplay unit 10 to display an augmented reality space. As in each of theabove examples, display objects such as the real object ROB, the virtualobject VOB, and the icon image Pi are displayed on the display unit 10by the display process performed by the display processing unit 23.

The switching processing unit 26 executes a process for switchingbetween the first mode and the second mode. The switching process isperformed on the basis of information obtained by image processingapplied to the captured image data as the first sensing data, forexample.

In a case where the finger of the user carrying the smartphone 1 can beidentified from the captured image data, for example, the switchingprocessing unit 26 performs a process for transitioning to the firstmode, on the basis of recognition of detection that the user attempts toconduct the first operation.

Note that the switching processing unit 26 may be included in thecontrol unit 9. In this case, the switching processing unit 26 of thecontrol unit 9 performs a process for switching between the first modeand the second mode, on the basis of information that is output to thecontrol unit 9 and that indicates that the image processing unit 4B hasrecognized the index finger IF as the predetermined portion of the user.

In a case where the user has conducted the first operation or the secondoperation, the operation target identification unit 27 performs aprocess for identifying, as a target of the conducted operation, thevirtual object VOB selected by the user as the target of the operation.

For example, the identification process for identifying the operationtarget is achieved on the basis of the captured image data as the firstsensing data output from the first sensing unit 2. Alternatively, theidentification process for identifying the operation target may beachieved with use of the distance data as the second sensing data outputfrom the second sensing unit 3. For example, the identification processmay be executed on the basis of the captured image data after narrowingdown to the virtual objects VOB located within a predetermined distancefrom the user. This is because the virtual object VOB designated as theoperation target for the user is often located near the user, andaccordingly, if the virtual object VOB located far away is identified asthe operation target, a process not intended by the user is highlylikely to be executed.

Each of example cases of the processes executed by the smartphone 1according to the third embodiment will be described herein.

<7-1. Fifth Case>

A fifth case is a case where the smartphone 1 executes a process fordetecting the index finger IF of the user, and a process for switchingbetween the first mode and the second mode according to a detectionresult.

FIG. 17 depicts an example of processes executed by the smartphone 1.Note that the image processing unit 4B and the control unit 9 willcollectively and simply be referred to as a “processing unit” as in theother cases described above.

In Step S201, the processing unit determines whether or not thedetection of the index finger IF of the user is already in progress. Aseries of processes depicted in FIG. 17 are executed every time capturedimage data is acquired, i.e., for each frame obtained by the imagesensor 5, for example. The determination process in Step S201 executedaccording to capture of a latest frame is a process for determiningwhether or not the index finger IF of the user has been detected incaptured image data acquired one frame before the latest frame.

In a case of determining that the index finger IF of the user has notbeen detected yet (determination “No”), the processing unit determinesin Step S202 whether or not the index finger IF of the user has beendetected in captured image data as the latest frame. In a case ofdetermination “Yes” in the determination process in Step S202, i.e., ina case where the new index finger IF of the user has been detected inthe latest frame, the processing unit performs a transition process fortransitioning to the first mode in Step S203.

Thereafter, the processing unit performs a process for disabling thesecond operation in Step S204. This process is a process for prohibitingacceptance of a touch operation for the display unit 10 of thesmartphone 1.

In Step S205, the processing unit performs a process for changing adisplay mode of the virtual object VOB and the icon image Pi. The firstoperation is an operation performed by the user for selecting thevirtual object VOB in consideration of a positional relation between thevirtual object VOB displayed on the display unit 10 and a tip of theindex finger IF of the user. In this case, there is a possibility thatthe display of the icon image Pi becomes an obstacle to the operation incomparison with the second operation for touching the virtual object VOBdisplayed on the display unit 10.

Accordingly, in the process in Step S205, the processing unit mayexecute a process for reducing the size of the icon image Pi or aprocess for hiding display of the icon image Pi.

This process will specifically be described with reference to FIGS. 18and 19 . FIG. 18 is a diagram depicting a state where the firstoperation is conducted to select, among virtual objects VOB6, VOB7, andVOB8, the virtual object VOB6 while the icon image Pi is displayed. Onthe other hand, FIG. 19 is a diagram depicting a state where the firstoperation is conducted while display of the icon image Pi is hidden.

In the manner described above, the operation for selecting the virtualobject VOB is facilitated by hiding display of the icon image Pi.

Meanwhile, in a case where the size of the virtual object VOB is smallon the display, there is a possibility that the second operation isdifficult to conduct. For dealing with such a case, a process forincreasing the size of the virtual object VOB may be executed in StepS205.

FIG. 18 depicts a state where the virtual objects VOB6, VOB7, and VOB8are displayed in sizes corresponding to respective distances. In thisstate, the virtual object VOB6 is displayed in a small size, andtherefore, the first operation for selecting the virtual object VOB6 isdifficult to conduct in some cases. Accordingly, it is considered toadopt a manner of display depicted in FIG. 19 where the sizes of thevirtual objects VOB6 and VOB7 are equivalent to the size of the virtualobject VOB8. In this manner, the operation for selecting the virtualobject VOB6 located farthest from the smartphone 1 is facilitated.

Note that the state depicted in FIG. 19 is a state where parts of thevirtual objects VOB are hidden behind the index finger IF of the user.This state is achieved by designating the regions hidden by the indexfinger IF of the user as occlusion regions and carrying out an occlusionprocess to hide display of the portions disposed in the occlusionregions in the virtual objects VOB.

Moreover, the state depicted in FIG. 19 is a state where parts of thethree virtual objects VOB6, VOB7, and VOB8 overlap with each other. Thearrangement may be adjusted in such a manner as to eliminate overlapbetween the virtual objects VOB in order to facilitate the operation forselecting the virtual object VOB.

After completion of the process in Step S205, the processing unit endsthe series of processes illustrated in FIG. 17 .

In addition, in a case of determining in Step S202 that the index fingerIF of the user has not been detected, i.e., in a case of determiningthat the state of no detection of the index finger IF of the usercontinues, the processing unit also ends the series of processesillustrated in FIG. 17 . In this case, transition to the first mode doesnot take place, and the second mode continues.

On the other hand, in a case of determining in Step S201 that thecurrent state is a state where detection of the index finger IF of theuser is already in progress, i.e., in a case where transition to thefirst mode has already been completed, the processing unit advances theprocess to Step S206 to perform a process for determining whether or notthe index finger IF of the user has been detected in the captured imagedata as the latest frame.

Here, in a case where the index finger IF of the user has been detected,i.e., in a case of determining that the state of detection of the indexfinger IF of the user continues, the processing unit ends the series ofprocesses illustrated in FIG. 17 while continuing the first mode.

On the other hand, in a case of determining in Step S206 that the indexfinger IF of the user has not been detected, i.e., in a case ofdetermining that the state of detection of the index finger IF of theuser has changed to a state of no detection, the processing unitperforms a transition process for transitioning to the second mode inStep S207.

The processing unit performs a process for enabling the second operationin Step S208, and changes a display mode in Step S209.

For changing of the display mode in Step S209, the processing unitexecutes a process for returning to the state where the icon image Pi isdisplayed in a large size or a process for returning to the state wherethe virtual objects VOB6 and VOB7 are displayed in small sizes (ornormal sizes). Accordingly, the display is changed from the statedepicted in FIG. 20 to the state depicted in FIG. 21 .

After completion of the process in Step S209, the processing unit endsthe series of processes illustrated in FIG. 17 .

Note that, while described with reference to FIG. 17 has been theexample which executes Step S204 and Step S205 after completion oftransition to the first mode in Step S203, one or some of the processesin Step S203, Step S204, and Step S205 may be configured to be executed.

Similarly, one or some of the processes in Step S207, Step S208, andStep S209 may be executed.

For example, in a case where the state of no detection of the indexfinger IF of the user is changed to the state of detection of the indexfinger IF of the user, only the process in Step S203 for transitioningto the first mode may be executed. In a case where the state ofdetection of the index finger IF of the user is changed to the state ofno detection of the index finger IF of the user, only the process inStep S207 for transitioning to the second mode may be executed.

In these cases, the process for detecting the index finger IF of theuser may be executed with a frequency different between the first modeand the second mode. Specifically, in the case of transition to thefirst mode, the frequency of execution of the process for detecting theindex finger IF of the user or the process for identifying the positionof the tip of the finger of the user may be raised to recognize movementof the index finger IF of the user with high accuracy. In the case oftransition to the second mode, the frequency of execution of the processfor detecting the index finger IF of the user may be lowered (e.g., oncefor every one-tenth of a second).

Alternatively, in the case where the state of no detection of the indexfinger IF of the user is changed to the state of detection of the indexfinger IF of the user, the process for transitioning to the first modein Step S203 and the process for disabling the second operation in StepS204 may be executed without executing the process in Step S205. In thecase where the state of detection of the index finger IF of the user ischanged to the state of no detection of the index finger IF of the user,the process for transitioning to the second mode in Step S207 and theprocess for enabling the second operation in Step S208 may be executedwithout executing the process in Step S209.

While the example which has the first mode for accepting the firstoperation and the second mode for accepting the second operation hasbeen described with reference to FIG. 17 , a different mode such as athird mode may further be provided.

For example, the first mode may be a mode for accepting only the firstoperation, the second mode may be a mode for accepting only the secondoperation, and the third mode may be a mode for accepting each of thefirst operation and the second operation.

The following configuration may be adopted. The third mode is adoptedfor operation in an initial state. In a case where imaging of the indexfinger IF of the user is identified on the basis of image processing forcaptured image data, the mode transitions to the first mode. In a casewhere a touch operation performed by the user for the display unit 10 isdetected, the mode transitions to the second mode. Thereafter, in a casewhere nothing is further detected any more, the mode may be configuredto return to the third mode.

In this configuration, erroneous input by the user is prevented by thetransition to the first mode and the second mode. Moreover, a standbystate for handling any operation by the user is produced by thetransition to the third mode.

In addition, registered data of the index finger IF of the user may beused for detection of the index finger IF of the user. For example, byregistering only the right and left index fingers IF of the userpossessing the smartphone 1, a possibility of erroneous detection ofthings that are similar to but different from the index fingers IF ofthis user and a possibility of erroneous detection of index fingers IFof other persons or other fingers (e.g., middle finger) can beeliminated. In this manner, a more appropriate process can be carriedout.

A modification of the fifth case will be described.

While described with reference to FIG. 19 has been the example where apart of the virtual object VOB is hidden behind the index finger IF ofthe user in a case where the index finger IF of the user is locatedwithin the angle of view, the real objects ROB and the virtual objectsVOB located in the occlusion regions hidden behind the index finger IFmay be vaguely displayed by performing a process for converting theindex finger IF of the user into a semitransparent finger.

Such a display process is achievable by a-blending using captured imagedata obtained before an occlusion is caused in an occlusion region andcaptured image data as a latest frame after the occlusion is caused, forexample.

Alternatively, a-blending may be carried out with use of image dataobtained from a three-dimensional model acquired beforehand.

<7-2. Sixth Case>

A sixth case is a case where the smartphone 1 executes a process fordetecting the first operation and the second operation conducted by theuser and a process performed in correspondence with the detectedoperation.

FIG. 22 depicts an example of processes executed by the smartphone 1.Note that the image processing unit 4B and the control unit 9 hereinwill collectively and simply be referred to as the “processing unit” asin the other cases described above.

In Step S301, the processing unit determines whether or not the firstoperation has been detected. In a case of determining in Step S302 thatthe first operation has been detected, the processing unit performs aprocess for identifying the virtual object VOB designated as a target ofthe first operation. For example, the virtual object VOB at which theindex finger IF of the user is pointing is identified on the basis ofcaptured image data as the first sensing data, i.e., image datadisplayed on the display unit 10. In the case of the example depicted inFIG. 19 , the virtual object VOB6 is identified as the virtual objectVOB corresponding to a selection target.

In Step S303, the processing unit executes a process corresponding tooperation contents of the first operation and the identified virtualobject VOB.

The operation contents of the first operation will be touched uponherein. The user is allowed to conduct multiple gesture operations asthe first operation in an augmented reality space presented by thesmartphone 1.

For example, various types of actions are adoptable, such as actions ofgrasping, pinching, hitting, placing on the palm, flicking, stroking,and turning over the virtual object VOB corresponding to the operationtarget.

For example, adoptable corresponding processes are a process for causingthe display unit 10 to display a web page associated with the realobject ROB on which the virtual object VOB is superimposed (hereinafterreferred to as a “target real object ROB”), a process for causing thedisplay unit 10 to display simplified information associated with thetarget real object ROB in such a form as a balloon, a process forchanging display of the virtual object VOB to hidden display, a processfor changing a display color, a shape, a size, or the like of thevirtual object VOB, a process for obtaining a coupon of a storeassociated with the target real object ROB, a process for registering aposition of a store as the target real object ROB, and a process forstarting an application (e.g., camera application) different from anapplication for displaying the augmented reality space. Moreover, forexample, the adoptable corresponding process is a process for changinglight emission intensity of the light emitting unit 7 included in thesecond sensing unit 3, to measure a distance to the target real objectROB.

The following situation is considered as an example of a light emissionintensity change of the light emitting unit 7. In a case where the realobject ROB on which the virtual object VOB designated as an operationtarget is superimposed is located at a distance longer than apredetermined distance, light emission intensity is increased to measurethe distance to this real object ROB.

The various corresponding processes described above may be executed inassociation with types of gesture operations as the first operation. Inother words, processes to be executed may be configured to changeaccording to a change of gesture operations.

Moreover, various types of the virtual objects VOB are adoptable. Theoperable types of the first operations and the corresponding processesmay be determined according to the types of the virtual objects VOB.

For example, if the virtual object VOB is an object associated with anadvertisement, a process for displaying a web page presenting details ofthe advertisement or a web page where an advertised product ispurchasable is allowed to be executed as the corresponding process. Inaddition, only the first operation for executing such a correspondingprocess may be made acceptable.

Further, the gestures and the corresponding processes allowed to beexecuted may vary according to not the types of the virtual objects VOB,but the stores as the real objects ROB on which the virtual objects VOBare superimposed. For example, the processing unit may be configured toaccept a gesture operation for moving a finger in a manner following anexternal shape of a logo or a character of the store as the real objectROB or a gesture operation for changing a shape of a hand in a mannerimitating the shape of the logo or the character. In addition, theprocessing unit may be configured to execute a process for acquiring acoupon associated with this store to perform the corresponding process.In other words, even if the virtual object VOB is of a similar typeindicating a signboard of a store, the gesture operation allowed to beexecuted may be varied according to the type of the store.

Note that multiple processes may be executed as the correspondingprocesses. For example, a process for acquiring position informationassociated with the virtual object VOB, in response to a specificgesture operation conducted for the virtual object VOB, a process forstarting an application as SNS (Social Networking Service), and aprocess for inputting, to a posted message column of this application,information for identifying the position of the virtual object VOB orinformation associated with a store name or the like may be executed asthe corresponding processes.

After completion of the process in Step S303, the processing unit endsthe series of processes illustrated in FIG. 22 .

In a case of determining in Step S301 that the first operation has notbeen detected, the processing unit determines in Step S304 whether ornot the second operation has been detected.

In a case of determining that the second operation has been detected,the processing unit performs, in Step S305, a process for identifyingthe virtual object VOB designated as an operation target. This processis a process for identifying the virtual object VOB displayed at aposition corresponding to a touch operation performed by the user on thedisplay unit 10.

After the virtual object VOB as the operation target is identified, theprocessing unit executes a process corresponding to operation contentsof the second operation and the identified virtual object VOB in StepS306.

As described above, various types of the touch operations as the secondoperation are adoptable, such as a single tap operation and a double tapoperation. Moreover, various types of the corresponding processes areadoptable as in the first operation. In addition, the acceptable secondoperation and the corresponding process may be determined according tothe type of the virtual object VOB.

Note that the corresponding process allowed to be executed may be aprocess different between the first operation and the second operation.As an example, the processing unit may be configured to accept anoperation for moving the virtual object VOB as the first operation, andprohibit acceptance of this operation as the second operation. Further,on the contrary, the processing unit may be configured to accept anoperation for displaying a web page associated with the virtual objectVOB as the second operation, and prohibit acceptance of this operationas the first operation.

As described above, the acceptable operation is varied for the firstoperation and the second operation. Accordingly, only an operationlikely to be performed by a human directly touching the virtual objectVOB is made acceptable as the first operation, and only an operationlikely to be performed by a human touching the virtual object VOBdisplayed on the display unit 10 is made acceptable as the secondoperation. In this manner, the user is allowed to perform intuitiveoperations, and therefore easily achieve separate use of the firstoperation and the second operation without consciousness.

8. Fourth Embodiment

A fourth embodiment is an example of a smartphone 1C which includes twoimage sensors.

FIG. 23 depicts a configuration example of the smartphone 1C.

Note that configurations similar to the corresponding configurations ofthe smartphone 1 in the first embodiment depicted in FIG. 2 are givenidentical reference signs, and will not be repeatedly described whereappropriate.

The smartphone 1C includes the first sensing unit 2, the second sensingunit 2, and a third sensing unit 41 as three sensing units.

The first sensing unit 2 includes a first image sensor 5C and a firstimaging optical system 6C. The first image sensor 5C and the firstimaging optical system 6C are configured similarly to the image sensor 5and the imaging optical system 6 in the first embodiment.

The second sensing unit 3 includes the light emitting unit 7 and thedistance sensor 8, and is configured similarly to the second sensingunit 3 in the first embodiment.

The third sensing unit 41 includes a second image sensor 42 and a secondimaging optical system 43. Accordingly, the first sensing unit 2 and thethird sensing unit 41 constitute two image sensors included in thesmartphone 1C.

The second image sensor 42 includes multiple pixels similarly to thefirst image sensor 5C. Moreover, the second imaging optical system 43 isconfigured similarly to the first imaging optical system 6C, andincludes multiple types of lenses and the like.

The first image sensor 5C of the first sensing unit 2 captures an imagein a facing direction of the surface of the smartphone 1C on the sideopposite to the side where the display unit 10 is provided, i.e., theback side of the display unit 10 as viewed from the user visuallyrecognizing the display unit 10, to thereby output captured image dataas the first sensing data. The sensing data output from the first imagesensor 5C herein is data obtained by imaging the rear side of thesmartphone 1C, and therefore will be referred to as rear captured imagedata.

The second image sensor 42 of the third sensing unit 41 captures animage in a facing direction of the display surface of the display unit10 of the smartphone 1C, i.e., the front side as viewed from the uservisually recognizing the display unit 10, to thereby output capturedimage data as third sensing data. The sensing data output from thesecond image sensor 42 herein is data obtained by imaging the front sideof the smartphone 1C, and therefore will be referred to as frontcaptured image data.

Other parts of the smartphone 1C are configured similarly to thecorresponding parts in the first embodiment, and therefore will not berepeatedly described in detail. Note that the smartphone 1C has theimage processing unit 4B having each of the functions depicted in FIG.16 .

As depicted in FIG. 16 , the image processing unit 4B includes the dataacquisition unit 21, the recognition processing unit 22, the displayprocessing unit 23, the switching processing unit 26, and the operationtarget identification unit 27.

The recognition processing unit 22 performs a process for recognizingthe index finger IF of the user. Either the captured image data or thedistance data is available for performing the process for recognizingthe index finger IF of the user. Note that the recognition processingunit 22 detects the index finger IF of the user by using both the rearcaptured image data and the front captured image data.

The switching processing unit 26 performs a process for switching to thefirst mode in a case of detection of the index finger IF on the basis ofthe rear captured image data. On the other hand, the switchingprocessing unit 26 performs a process for switching to the second modein a case of detection of the index finger IF on the basis of the frontcaptured image data.

The data acquisition unit 21, the display processing unit 23, and theoperation target identification unit 27 have configurations similar tothe corresponding configurations described in the above example, andtherefore will not be repeatedly described.

An example of processes executed by the image processing unit 4B and thecontrol unit 9 of the smartphone 1 according to the fourth embodimentwill be described with reference to FIG. 24 . Note that the imageprocessing unit 4B and the control unit 9 herein will collectively andsimply be referred to as a “processing unit” as in the other examplesdescribed above. In addition, processes similar to the processes in FIG.17 are given identical reference signs, and will not be repeatedlydescribed.

In Step S211, the processing unit determines whether or not the indexfinger IF of the user has been detected. In a case where the indexfinger IF of the user has been detected on the basis of the rearcaptured image data or the front captured image data, the processproceeds to Step S212. Note that the index finger IF detected herein maybe any of the registered index fingers IF as own fingers of the user.

In a case of determining that the index finger IF of the user has beendetected, the processing unit determines in Step S212 whether or notthis detection has been made on the basis of the rear captured imagedata. In a case where the index finger IF of the user has been detectedon the basis of the rear captured image data, the processing unitadvances the process to Step S213 to determine whether or not thecurrent mode is the first mode.

In a case where the current mode is not the first mode, i.e., in a casewhere the current mode is the second mode or where no mode is set, theprocessing unit performs a process for transitioning to the first modein Step S203, a process for disabling the second operation in Step S204,and a process for changing a display mode in Step S205. As described inthe third embodiment, one or some of the processes in Step S203, StepS204, and Step S205 may be executed.

In a case where the index finger IF of the user has been detected butnot on the basis of the rear captured image data, i.e., in a case ofdetermination “No” in Step S212, it is considered that this detectionhas been made on the basis of the front captured image data. In thiscase, the processing unit performs a process for determining in StepS214 whether or not the current mode is the second mode.

In a case where the current mode is not the second mode, the processingunit performs a process for transitioning to the second mode in StepS207, a process for enabling the second operation in Step S208, and aprocess for changing the display mode in Step S209. As described in thethird embodiment, one or some of the processes in Step S207, Step S208,and Step S209 may be executed.

On the other hand, in the case where the current mode is the secondmode, i.e., in a case of determination “Yes” in Step S214, theprocessing unit ends a series of processes illustrated in FIG. 24 .

In a case where it is determined in Step S211 that the index finger IFof the user has not been detected, there is a possibility that the userconducts the second operation. Accordingly, the processing unit performsa process for enabling the second operation in Step S215. However, theprocess in Step S215 may be skipped in a case where the second operationhas already been enabled. Meanwhile, if the process for disabling thesecond operation is not executed at the time of transition to the firstmode, the process in Step S215 itself is unnecessary.

9. Summary

As described in the foregoing various examples, the smartphone 1 (1A,1C) as an image processing apparatus includes the data acquisition unit21 that acquires first sensing data that is image sensing data andsecond sensing data that is sensing data of a type different from thefirst sensing data, and the display processing unit 23 that performs adisplay process for displaying an augmented reality space formed byaugmenting a real space with use of the first sensing data and thesecond sensing data.

According to this configuration, the augmented reality space can bedisplayed on the basis of information not acquirable from the imagesensing data.

For example, information associated with a step or the like difficult torecognize from image data can be extracted from the second sensing data,and displayed on the real space.

As described with reference to FIG. 3 , FIGS. 4 to 8 , and otherfigures, the display processing unit 23 of the smartphone 1 (1A, 1C) mayperform, by using the second sensing data, a display process forsuperimposing the virtual object VOB on the real object ROB recognizedas an object present in the real space.

The superimposed display of the virtual object VOB on the real objectROB achieves display of an augmented reality space where the virtualobject VOB is arranged without giving a sense of strangeness.

Moreover, by displaying superimposed information associated with thereal object ROB as the virtual object VOB, information necessary for theuser can be provided without giving a sense of strangeness.

As described with reference to FIG. 2 and other figures, the secondsensing data of the smartphone 1 (1A, 1C) may be sensing data from whichdistance information (distance data) associated with the real object ROBis extractable.

According to this configuration, the display mode of the virtual objectVOB can be varied according to the distance information.

For example, the size of the virtual object VOB to be superimposed canbe varied according to the distance of the real object ROB, or displayof the virtual object VOB to be superimposed can be made moreconspicuous as the real object ROB is located closer.

As described in the first case with reference to FIG. 4 and otherfigures, the display processing unit 23 of the smartphone 1 (1A, 1C) mayperform a display process for issuing an alert regarding a step detectedwith use of the distance information (distance data).

According to this configuration, a notification of informationassociated with a step or the like not recognized by the user can beissued as the virtual object VOB.

For example, in a case where the image processing apparatus having thisconfiguration is a portable terminal device such as the smartphone 1(1A, 1C) including the image processing unit 4 (4A, 4B), thisconfiguration contributes to improvement of safety for the user walkingwhile operating the portable terminal device.

As described in the third case with reference to FIG. 6 and otherfigures, the display processing unit 23 of the smartphone 1 (1A, 1C) maydetermine the size of the virtual object VOB superimposed on the realobject ROB, according to the distance of the real object ROB.

According to this configuration, the size of the virtual object VOB isdetermined according to the distance from the user.

Accordingly, this configuration achieves superimposed display of thevirtual objects VOB without giving a sense of strangeness.

As described in the second embodiment with reference to FIG. 10 , thesecond sensing data of the smartphone 1 (1A, 1C) may be sensing data onthe basis of which the material of the real object ROB is identifiable.The display processing unit 23 may perform a display process forsuperimposing the virtual object VOB on the real object ROB whosematerial is identified.

For example, the sensor for outputting the second sensing data is thepolarization sensor 31, a multispectral sensor, or the like.

With use of the second sensing data on the basis of which the materialis identifiable, a display mode of the virtual object VOB is adoptableaccording to the material. Specifically, by adopting such an expressionsuited for the material of the real object ROB to improve reality of thevirtual object VOB, a sense of immersion of the user into the augmentedreality space can be enhanced.

As described in the second embodiment with reference to FIG. 12 andother figures, the smartphone 1 (1A, 1C) may include the self-positionestimation unit 25 that estimates a self-position. The self-positionestimation unit 25 may correct the self-position on the basis ofmaterial information associated with the real object ROB.

According to this configuration, highly accurate self-positioninformation can be obtained.

Accordingly, optimum information can be superimposed and displayed asthe virtual object VOB on the basis of highly accurate self-positioninformation. For example, by estimating floor information or the like,which is difficult to identify on the basis of GNSS information or thelike, from the second sensing data, the virtual object VOB containingappropriate information corresponding to a floor can be superimposed anddisplayed.

As described in the second embodiment with reference to FIG. 12 andother figures, the display processing unit 23 of the smartphone 1 (1A,1C) may perform a display process for displaying the virtual object VOBto be superimposed, in a mode different between the real object ROBwhose material identified with use of the second sensing data is atransparent material and the real object ROB whose material identifiedwith use of the second sensing data is not a transparent material.

For example, in a case where the first sensing data that is the imagesensing data is used, there is a possibility that the real object ROBsuch as a door including glass and a window is difficult for the user torecognize. In such a case, the real object ROB which is a glass membercan be recognized with use of the polarization sensor 31, amultispectral sensor, or the like, thereby allowing the virtual objectVOB to be superimposed and displayed on the glass member.

The superimposed display of the virtual object VOB on the glass memberallows the user to recognize the presence of the glass member, andtherefore contributes to improvement of safety. Moreover, for example,semitransparent display of the virtual object VOB superimposed anddisplayed on the glass member produces a state allowing recognition ofthe presence of another real object ROB located on the back side of theglass member as well. In other words, this configuration can create sucha state allowing visual recognition of a target object by both a userdesiring to visually recognize the virtual object VOB superimposed anddisplayed on the glass member and a user desiring to visually recognizethe real object ROB located on the back side of the glass member.Accordingly, user-friendliness improves.

As described in the second embodiment with reference to FIG. 10 andother figures, the second sensing data of the smartphone 1 (1A, 1C) maybe sensing data on the basis of which a polarized state of lightreflected from the real object ROB is identifiable.

Specifically, the second sensing data is sensing data output from thepolarization sensor 31.

In this manner, the advantageous effects described above can be obtainedwith use of the polarization sensor 31.

As described in the fourth case with reference to FIG. 7 , FIG. 8 , andother figures, the display processing unit 23 of the smartphone 1 (1A,1C) may perform a display process for superimposing the virtual objectVOB on the real object ROB identified as a part of a human body on thebasis of the second sensing data.

For example, some persons do not desire to appear in an image or avideo. Particularly, considering issues of privacy, it is not preferableto store or stream an image captured in a state where persons areidentifiable. According to this configuration, a part of a human body,such as a face, is identified, and the virtual object VOB can besuperimposed and displayed on the identified part of the human body.

In this manner, identification of a person within a captured image canbe prohibited, and therefore, privacy protection of a third party can beguaranteed.

As described in the second embodiment, the first sensing data and thesecond sensing data of the smartphone 1 (1A, 1C) may be generated on thebasis of signals output from an identical sensor.

For example, not only image sensing data but also distance data can beacquired from data output from a CMOS sensor, in some cases. Moreover,image sensing data and distance data can be acquired in a case of use ofan image sensor carrying a mixture of image plane phase-differencepixels and color pixels in red, blue, and green.

By extracting the first sensing data and the second sensing data fromthe one sensor as described above, cost reduction and size reduction ofthe smartphone 1 (1A, 1C) as the image processing apparatus are bothachievable.

As described in the second embodiment with reference to FIG. 14 andother figures, the smartphone 1 (1A, 1C) may include the switchingprocessing unit 26 that switches between the first mode for acceptingthe first operation for the virtual object VOB and the second mode foraccepting the second operation for the virtual object VOB. The switchingprocessing unit 26 may switch to the first mode in a case where apredetermined portion of a user (e.g., the index finger IF) isrecognized within the augmented reality space with use of the firstsensing data or the second sensing data.

For example, the first process is executed in a case where a finger ofthe user (e.g., index finger IF) is recognized on the basis of thesecond sensing data from which distance information is extractable.

In this manner, the first process can be executed with use of thedistance information associated with the finger of the user.

As described in the third embodiment with reference to FIG. 14 , FIG. 15, and other figures, in the smartphone 1 (1A, 1C), the first mode may bea mode for accepting an operation achieved for the virtual object VOBaccording to movement of the predetermined portion (e.g., index fingerIF) in the augmented reality space, and the second mode may be a modefor accepting a touch operation performed by the user for a displayregion of the virtual object VOB on the display unit 10.

For example, in a case where a virtual touch operation performed by afinger of the user (e.g., index finger IF) for the virtual object VOB inthe augmented reality space is detected by a ranging sensor or the like,transition to the first mode is achieved. In a case where a touchoperation for the image region of the virtual object VOB displayed onthe display unit is detected, transition to the second mode is achieved.

In this manner, the user is allowed to switch the operation mode inconsideration of ease of operation or the like. In other words,operability improves.

As described in the fifth case with reference to FIGS. 18 to 21 andother figures, the display processing unit 23 of the smartphone 1 (1A,1C) may designate a different display mode of the display object(virtual object VOB, icon image Pi) between the first mode and thesecond mode.

The display object is an object displayed on the display unit. Forexample, the display object includes not only the virtual object VOBarranged in the augmented reality space, but also the icon image Pi andthe like not arranged in the augmented reality space. The example whichvaries the display mode of the display object includes an example whichchanges a display size, an example which switches between display andnon-display for each display object, and other examples.

For example, in a case of execution of an operation for directlytouching the virtual object VOB arranged in the augmented reality space,visibility of the virtual object VOB may be lowered by an operation icondisplayed on the display unit 10. Accordingly, operability can be raisedby performing a display process for hiding display of the operation iconin the first mode.

Specifically, an optimum UI (User Interface) for the first mode may bedifferent from an optimum UI for the second mode. In this case,operability can be raised by designating the different display mode ofthe display object between the first mode and the second mode.

As described in the fifth case with reference to FIGS. 18 to 21 andother figures, the display processing unit 23 of the smartphone 1 (1A,1C) may perform a display process for designating a different size ofthe virtual object VOB between the first mode and the second mode.

For example, an optimum UI for the first mode may be different from anoptimum UI for the second mode. In this case, a different size of thevirtual object VOB is designated for the first mode and the second mode.Specifically, in a case where an operation for the virtual object VOB isdetected in the augmented reality space, the virtual object VOB isdisplayed in a large size to improve visibility, for the user, of letterinformation or the like displayed as the virtual object VOB. Meanwhile,in a case where a touch operation or the like for the display unit 10 isdetected, this operation is recognized as a selection operation forselecting the virtual object VOB, and a process for displaying a webpage associated with the virtual object VOB provided as an advertisementobject or the like is executed, for example.

In the manner described above, the size of the virtual object VOB isvaried according to the operation mode to improve operability for theuser.

As described in the fifth case with reference to FIG. 18 , FIG. 19 , andother figures, the display processing unit 23 of the smartphone 1 (1A,1C) may perform an occlusion process for the virtual object VOB on thebasis of distance information (distance data) associated with thepredetermined portion (e.g., index finger IF).

For example, in a case where the predetermined portion is a finger ofthe user and is located at a short distance, an occlusion process isperformed to hide the virtual object VOB behind the finger of the user.

In this manner, the user is allowed to visually recognize the augmentedreality space without feeling a sense of strangeness.

As described in the second embodiment with reference to FIG. 16 andother figures, the smartphone 1 (1A, 1C) may include the operationtarget identification unit 27 that identifies the virtual object VOBdesignated as a target of an operation performed by the predeterminedportion (e.g., index finger IF), with use of the first sensing data in acase where the predetermined portion is recognized.

It is possible that the user operates the virtual object VOB whileviewing the virtual object VOB and an image of his or her finger whichare displayed on the display unit 10. In such a case, the virtual objectVOB corresponding to the operation target is identified according to apositional relation between the image of the finger and the virtualobject VOB on the display unit 10.

In this manner, the user is allowed to select the virtual object VOBcorresponding to the operation target by an intuitive operation.Accordingly, operability improves.

As described in the fourth embodiment with reference to FIG. 23 and FIG.24 , the switching processing unit 26 of the smartphone 1 (1A, 1C) mayswitch to the second mode in a case where the predetermined portion(e.g., index finger IF) is detected on the basis of sensing data (frontcaptured image data) obtained by imaging in a direction opposite to animaging direction of an imaging unit (first sensing unit 2) that obtainsthe first sensing data (rear captured image data).

In such a case, it is estimated that the predetermined portion of theuser is located before the display unit 10, and it is therefore assumedthat the first operation is not likely to be conducted. Accordingly, itis considered that switching to the second mode for accepting the secondoperation is most appropriate. After switching to the second mode, suchprocesses as the process for detecting the predetermined portion of theuser on the basis of the second sensing data are suspended.

In this manner, the necessity of performing various processes assumed tobe executed by the smartphone 1 in the first mode is eliminated.Accordingly, reduction of a processing load on the smartphone 1 isachievable.

Note that the fourth embodiment described above has been the examplewhere the smartphone 1C performs the process for detecting the indexfinger IF as the predetermined portion of the user with use of thecaptured image data output from the first sensing unit 2 and thedistance data output from the second sensing unit 3 and the displayprocess corresponding to the detected index finger IF. In other words,the example where the smartphone 1C has the three sensing units has beendescribed.

The smartphone 1C in a different mode may include a sensing unit whichoutputs two types of captured image data, and omit a sensing unit whichoutputs distance data.

In other words, the image processing apparatus may be configured toexecute image processing for an augmented reality space by acquiring onetype of sensing data (captured image data) from two camera units,instead of executing image processing for an augmented reality space onthe basis of two types of sensing data (e.g., captured image data anddistance data).

The image processing apparatus described above can have the followingconfiguration.

An image processing apparatus including:

-   -   a data acquisition unit that acquires first captured image data        output from a first imaging unit for imaging in a first        direction and second captured image data output from a second        imaging unit for imaging in a direction opposite to the first        direction; and    -   a display processing unit that performs a display process for        displaying an augmented reality space formed by augmenting a        real space with use of the first captured image data and the        second captured image data.

The display processing unit executes a first display process as thedisplay process in a case where a predetermined portion of a user isdetected on the basis of the first captured image data.

The display processing unit executes a second display process as thedisplay process in a case where the predetermined portion is detected onthe basis of the second captured image data.

The first imaging unit herein refers to the first sensing unit 2, suchas a rear camera, included in the smartphone 1C, while the secondimaging unit refers to the third sensing unit 41, such as a frontcamera, included in the smartphone 1C. Moreover, the first directionrefers to a facing direction of the surface of the smartphone 1C on theside opposite to the side where the display unit 10 is provided, whilethe second direction is a facing direction of the display unit 10.

Further, for example, the first process refers to a process fortransitioning to the first mode, a process for disabling the secondoperation, or a process for changing the display mode such as theprocess in Step S205.

In addition, for example, the second process refers to a process fortransitioning to the second mode, a process for enabling the secondoperation, or a process for changing the display mode such as theprocess in Step S209.

As obvious from above, the advantageous effects described above can beobtained without using distance data.

Note that advantageous effects to be offered are not limited to thosedescribed in the present description by way of example. Otheradvantageous effects may further be produced.

In addition, the examples described above may be combined in anymanners. The various operational advantages described above cansimilarly be obtained even from various combinations of the aboveexamples.

10. Present Technology

The present technology can also adopt the following configurations.

(1)

An image processing apparatus including:

-   -   a data acquisition unit that acquires first sensing data that is        image sensing data and second sensing data that is sensing data        of a type different from the first sensing data; and    -   a display processing unit that performs a display process for        displaying an augmented reality space formed by augmenting a        real space with use of the first sensing data and the second        sensing data,    -   in which the display processing unit performs, by using the        second sensing data, a display process for superimposing a        virtual object on a real object recognized as an object present        in the real space.        (2)

The image processing apparatus according to (1) above, in which thesecond sensing data is sensing data from which distance informationassociated with the real object is extractable.

(3)

The image processing apparatus according to (2) above, including:

-   -   a switching processing unit that switches between a first mode        for accepting a first operation for the virtual object and a        second mode for accepting a second operation for the virtual        object,    -   in which the switching processing unit switches to the first        mode in a case where a predetermined portion of a user is        recognized within the augmented reality space by using the first        sensing data or the second sensing data.        (4)

The image processing apparatus according to (3) above,

-   -   in which the first mode is a mode for accepting an operation        performed for the virtual object according to movement of the        predetermined portion in the augmented reality space, and    -   the second mode is a mode for accepting a touch operation        performed by the user for a display region of the virtual object        on a display unit.        (5)

The image processing apparatus according to (4) above, in which thedisplay processing unit designates a different display mode of a displayobject between the first mode and the second mode.

(6)

The image processing apparatus according to (5) above, in which thedisplay processing unit performs a display process for designating adifferent size of the virtual object between the first mode and thesecond mode.

(7)

The image processing apparatus according to any one of (3) to (6) above,in which the display processing unit performs an occlusion process forthe virtual object on the basis of distance information associated withthe predetermined portion.

(8)

The image processing apparatus according to any one of (3) to (7) above,including:

-   -   an operation target identification unit that identifies the        virtual object designated as a target of an operation performed        by the predetermined portion, with use of the first sensing data        in a case where the predetermined portion is recognized.        (9)

The image processing apparatus according to any one of (3) to (8) above,in which the switching processing unit switches to the second mode in acase where the predetermined portion is detected on the basis of sensingdata obtained by imaging in a direction opposite to an imaging directionof an imaging unit that obtains the first sensing data.

(10)

The image processing apparatus according to any one of (2) to (9) above,in which the display processing unit performs a display process forissuing an alert regarding a step detected with use of the distanceinformation.

(11)

The image processing apparatus according to any one of (2) to (10)above, in which the display processing unit determines a size of thevirtual object to be superimposed on the real object, according to adistance of the real object.

(12)

The image processing apparatus according to any one of (1) to (11)above,

-   -   in which the second sensing data is sensing data on the basis of        which a material of the real object is identifiable, and    -   the display processing unit performs a display process for        superimposing the virtual object on the real object whose        material is identified.        (13)

The image processing apparatus according to (12) above, including:

-   -   a self-position estimation unit that estimates a self-position,    -   in which the self-position estimation unit corrects the        self-position on the basis of material information associated        with the real object.        (14)

The image processing apparatus according to any one of (12) to (13)above, in which the display processing unit performs a display processfor displaying the virtual object to be superimposed, in a modedifferent between the real object whose material identified with use ofthe second sensing data is a transparent material and the real objectwhose material identified with use of the second sensing data is not atransparent material.

(15)

The image processing apparatus according to any one of (12) to (14)above, in which the second sensing data is sensing data on the basis ofwhich a polarized state of light reflected from the real object isidentifiable.

(16)

The image processing apparatus according to any one of (12) to (15)above, in which the display processing unit performs a display processfor superimposing the virtual object on the real object identified as apart of a human body with use of the second sensing data.

(17)

The image processing apparatus according to any one of (2) to (11)above, in which the first sensing data and the second sensing data aregenerated on the basis of signals output from an identical sensor.

(18)

An image processing method causing a computer device to execute:

-   -   a process that acquires first sensing data that is image sensing        data and second sensing data that is sensing data of a type        different from the first sensing data; and    -   a display process for displaying an augmented reality space        formed by augmenting a real space with use of the first sensing        data and the second sensing data.        (19)

An image processing system including:

-   -   a display unit that is capable of displaying an image;    -   a first sensing unit that acquires first sensing data that is        image sensing data;    -   a second sensing unit that acquires second sensing data that is        sensing data of a type different from the first sensing data;        and    -   a display processing unit that performs a display process for        causing the display unit to display an augmented reality space        formed by augmenting a real space with use of the first sensing        data and the second sensing data.

REFERENCE SIGNS LIST

-   -   1, 1A, 1C: Smartphone    -   2: First sensing unit (first imaging unit)    -   10: Display unit    -   21: Data acquisition unit    -   23: Display processing unit    -   25: Self-position estimation unit    -   26: Switching processing unit    -   27: Operation target identification unit    -   41: Third sensing unit (second imaging unit)    -   ROB: Real object    -   VOB, VOB1, VOB2, VOB3, VOB4, VOB5, VOB6, VOB7, VOB8: Virtual        object (display object)    -   Pi: Icon image (display object)    -   IF: Index finger (predetermined portion)

1. An image processing apparatus comprising: a data acquisition unitthat acquires first sensing data that is image sensing data and secondsensing data that is sensing data of a type different from the firstsensing data; and a display processing unit that performs a displayprocess for displaying an augmented reality space formed by augmenting areal space with use of the first sensing data and the second sensingdata, wherein the display processing unit performs, by using the secondsensing data, a display process for superimposing a virtual object on areal object recognized as an object present in the real space.
 2. Theimage processing apparatus according to claim 1, wherein the secondsensing data is sensing data from which distance information associatedwith the real object is extractable.
 3. The image processing apparatusaccording to claim 2, comprising: a switching processing unit thatswitches between a first mode for accepting a first operation for thevirtual object and a second mode for accepting a second operation forthe virtual object, wherein the switching processing unit switches tothe first mode in a case where a predetermined portion of a user isrecognized within the augmented reality space by using the first sensingdata or the second sensing data.
 4. The image processing apparatusaccording to claim 3, wherein the first mode is a mode for accepting anoperation performed for the virtual object according to movement of thepredetermined portion in the augmented reality space, and the secondmode is a mode for accepting a touch operation performed by the user fora display region of the virtual object on a display unit.
 5. The imageprocessing apparatus according to claim 4, wherein the displayprocessing unit designates a different display mode of a display objectbetween the first mode and the second mode.
 6. The image processingapparatus according to claim 5, wherein the display processing unitperforms a display process for designating a different size of thevirtual object between the first mode and the second mode.
 7. The imageprocessing apparatus according to claim 3, wherein the displayprocessing unit performs an occlusion process for the virtual object ona basis of distance information associated with the predeterminedportion.
 8. The image processing apparatus according to claim 3,comprising: an operation target identification unit that identifies thevirtual object designated as a target of an operation performed by thepredetermined portion, with use of the first sensing data in a casewhere the predetermined portion is recognized.
 9. The image processingapparatus according to claim 3, wherein the switching processing unitswitches to the second mode in a case where the predetermined portion isdetected on a basis of sensing data obtained by imaging in a directionopposite to an imaging direction of an imaging unit that obtains thefirst sensing data.
 10. The image processing apparatus according toclaim 2, wherein the display processing unit performs a display processfor issuing an alert regarding a step detected with use of the distanceinformation.
 11. The image processing apparatus according to claim 2,wherein the display processing unit determines a size of the virtualobject to be superimposed on the real object, according to a distance ofthe real object.
 12. The image processing apparatus according to claim1, wherein the second sensing data is sensing data on a basis of which amaterial of the real object is identifiable, and the display processingunit performs a display process for superimposing the virtual object onthe real object whose material is identified.
 13. The image processingapparatus according to claim 12, comprising: a self-position estimationunit that estimates a self-position, wherein the self-positionestimation unit corrects the self-position on a basis of materialinformation associated with the real object.
 14. The image processingapparatus according to claim 12, wherein the display processing unitperforms a display process for displaying the virtual object to besuperimposed, in a mode different between the real object whose materialidentified with use of the second sensing data is a transparent materialand the real object whose material identified with use of the secondsensing data is not a transparent material.
 15. The image processingapparatus according to claim 12, wherein the second sensing data issensing data on a basis of which a polarized state of light reflectedfrom the real object is identifiable.
 16. The image processing apparatusaccording to claim 12, wherein the display processing unit performs adisplay process for superimposing the virtual object on the real objectidentified as a part of a human body with use of the second sensingdata.
 17. The image processing apparatus according to claim 2, whereinthe first sensing data and the second sensing data are generated on abasis of signals output from an identical sensor.
 18. An imageprocessing method causing a computer device to execute: a process thatacquires first sensing data that is image sensing data and secondsensing data that is sensing data of a type different from the firstsensing data; and a display process for displaying an augmented realityspace formed by augmenting a real space with use of the first sensingdata and the second sensing data.
 19. An image processing systemcomprising: a display unit that is capable of displaying an image; afirst sensing unit that acquires first sensing data that is imagesensing data; a second sensing unit that acquires second sensing datathat is sensing data of a type different from the first sensing data;and a display processing unit that performs a display process forcausing the display unit to display an augmented reality space formed byaugmenting a real space with use of the first sensing data and thesecond sensing data.